DOCUMENT RESUME 



ED 389 264 



IR 017 466 



AUTHOR 
TITLE 



PUB DATE 
NOTE 



PUB TYPE 



Pugh, Richard C. ; And Others 

Interactive Television in Distance Education Setting: 
Formative Evaluation of "Adapting Instructional 
Techniques ♦ " 
Oct 95 

lip,; Paper presented at the Annual Meeting of the 
Midwestern Educational Research Association (Chicago, 
IL, October 1995) . 

Reports - Evaluative/Feasibility (142) — 
Speeches/Conference Papers (150) — Tests/Evaluation 
Instrument s (160) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIEP 



MF01/PC01 Plus Postage* 

*Advance Organizers; *Distance Education; Evaluation 

Methods ; Higher Education; ''Instruct ional 

Ef f ect iveness ; * Interact ive Televis ion; Model s ; 

Program Development; '''Teaching Methods; ''Videotape 

Recordings 

Prototypes 



ABSTRACT 

"Adapting Instructional Techniques 11 was developed at 
Indiana University to improve the performance of teachers in 
classrooms equipped for interactive audio or video in a distance 
education setting. The formative evaluation of this videotape is 
described. The tape suggested modifications for five common 
instructional techniques used in conventional classrooms: (1) 
demonstration; (2) discussion; (3) group work sessions; (4) lecture; 
and (5) question and answer. Three researchers reviewed the initial 
version of the tape and submitted it for review by eight experienced 
distance education instructors. The third phase of evaluation 
consisted of a survey review completed by eight experienced school 
teachers, who were contemplating offering a distance education 
course. The prototype version was judged favorably by most 
eval.uators, but some specific suggestions for improvement were given 
that included use of an advance organizer, more explication, more 
examples, and improved clarity and legibility of the graphics. An 
appendix contains the survey instrument. (SLD) 
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Methods 

Methods for the development of the videotapes for this study included (1 ) selecting classroom 
episodes from regular classroom instruction, (2) identifying alternate methods for handling the episodes 
for a distance education class, (3) developing video vignettes to illustrate alternative methods for 
handling the episodes in a distance education setting and recording them on videotape, and (4) 
conducting a formative evaluation of the videotapes. 

Five common instructional techniques which are used in conventional classroom settings were 
identified. An emphasis was placed on instructional moves within these instructional techniques which 
tended to "engage" the student. These instructional techniques were derived from materials produced 
by the Indiana University School of Nursing (Billings, 1 992) and from workshop materials developed 
by the Indiana University Center for Excellence in Education (1994). The five, selected techniques 
were (1) demonstration, (2) discussion, (3) group work sessions, (4) lecture, and (5) question and 
answer. 
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Evaluation sources 

Recommended procedures for the formative evaluation of the development of instructional 
materials have been reported by Dick and Carey (1990). These recommendations helped formulate 
a frame of reference for the formative evaluation phases of the video footage in this project. Three 
basic phases were followed. 

Information source for phase one. The first phase was a one-to-one or clinical evaluation of the 
video footage. The three investigators of this study participated directly in this initial phase of the 



formative evaluation. All three are experienced faculty at Indiana University and have been centrally 
involved in the technical and instructional support of the distance education classrooms for the past 
two years. 

Information source for phase two. The second phase of formative evaluation for this project 
consisted of a review by an experienced group of eight distance education instructors, all of whom had 
taught at least one course using these technologies. The prototype version of the videotape which 
emerged from phase one was used for this review. This phase generated evidence for the content 
validity of the videotape. 

Information source for phase three. The third phase of the formative evaluation consisted of a 
review by teachers who were representative of the target population of distance education instructors. 
The same prototype version of the videotape developed during phase one was used for this review. 
Eight, experienced school teachers who were contemplating offering a distance education course using 
these technologies reviewed the videotape. This phase served as a preliminary field test. 
Instrument. A Survey of Perceptions instrument (See Appendix) was developed and administered 
during phases two and three to all respondents following their review of Adapting Instructional 
Techniques using a written response format. The Survey instrument was based on formative 
evaluation procedures described by Dick and Carey (1990). A summary of the information from the 
administration of this survey to the experienced Distance Education Practitioner group and the 
Consumer group is reported in Table 1 . 

Information received during phase one 

Development of script. For each of the five selected instructional techniques, a comprehensive set 
of classroom instructional moves in a regular classroom was identified. A judgment was made about 
whether each instructional move was the same or different in a distance education setting. If it was 
judged to be different a revised statement of the instructional move was made. These steps were 
reviewed by an experienced distance education instructor and revisions were made in the statement 



of instructional moves for each of the five instructional techniques and a final statement was prepared. 
From these statements of moves, a script was developed for each instructional technique. Based on 
the five scripts, video footage of simulations of various instructional moves was shot during the 
summer, 1995, in both a regular classroom setting as well as a distance education setting. 
Development of prototype version. During the first phase of the formative evaluation, the one-to- 
one phase, the most obvious errors were identified and removed. This was conducted by direct 
interaction between the investigators and the experienced distance education instructor. During this 
phase, it was requested that the distance education instructor point out any reactions he had with the 
tape. It was judged that excellent rapport was established as evidenced by a very open interaction 
during the review of the footage with the investigators. From this review revised footage was 
assembled into an initial videotape which underwent four iterations of production. The current version 
of the videotape reflects these modifications. 

Information received during phases two and three 

Formative evaluation during phases two and three continued to include the collection of information 
to improve the effectiveness of Adapting Instructional Techniques. The purposes of formative 
evaluation during phases two and three were (1) to collect information from members of an 
experienced Distance Education Practitioner group concerning the content validity of Adapting 
Instructional Techniques and (2) to collect fnformation from a Consumer group which served the 
purpose of a preliminary field test. In essence, Adapting Instructional Techniques was developed to 
improve the performance of teachers in interactive audio/video equipped classrooms in a distance 
education setting. 

Table 1 contains a summary of the responses to the Survey of Perceptions by the Consumer 
group members and by the experienced Distance Education Practitioners group members. The 
frequency of each response is reported in parentheses after each response. 
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Table 1 . Item-by-Group Summary of Responses 



Survey of Perceptions 
Item 


Responses from preliminary 
field test-Consumer Group 
<N«8) 


Responses from 
Distance Education 
Practitioner Group (N~8) 


Is the initial motivation 
material interesting? 


Yes (6) 
Somewhat (2) 


Yes (4) 
Somewhat (2) 
Not found (2) 


Are the objectives clear and 
interesting as well as 
relevant? 


Yes (o) 

Introduction needs clarity (2) 


Yes (b) 

Part il needs elaboration (2) 
Part 1 needs context (1) 


A f . Z t 1 |t | — A. Till ■ jfciafl#H ml 

What skills that are illustrated 
in the videotape do you 
already possess? 


Regular classroom teaching 
techniques (8) 
Distance education (0) 


Regular classroom teaching 
techniques (7) 
Distance Education (7) 
No response (1) 


Does the videotape provide 
adequate vocabulary? 


Yes (5) 

No response (3) 


Yes (3) 

Too much jargon (2) 
No response (2) 


Does the videotape provide 
adequate pacing? 


Yes (4) 
Too slow (2) 
No response (2) 


Yes (3) 
Too slow (3) 
No response (2) 


uoes in© vioeouiptf provide 
adequate chunk size? 


Vac IA\ 

Needs to be longer (2) 
No response (2) 


Vac (41 

Needs to be longer (2) 
No response (2) 


Does the videotape provide 
aaequaie cianiy ot 
descriptions? 


Yes (4) 

iho response i*fi 


Yes (4) 

uiiToronuoo HOI CIBal 

between regular and distance 
ed (4) 


Does the videotape provide 
adequate clarity of examples? 


Yes (5) 

No response \o) 


Yes (3) 

Aaatuonai examples (oj 

Mixed (1) 

No response (1) 


Does the videotape provide 
adequate sequence of 
content and activities? 


Yes (4) 

No response (4) 


Vac 101 

Yes 

Problems with sequence (4) 
No response (2) 


UQB5 II 18 VlUUUloptl (JIUVIUU 

adequate interest level? 


Y«c (41 

T US \*rf 

Too many examples (2) 
No response (2) 


Nrt rosnnnQA (41 

Needs to be improved (4) 


Does the videotape provide 
adequate motivation? 


Yes (2) 

Suggestions for improvement 
(2) 

No response (4) 


YestD 

Suggestions for improvement 
(4) 

No response (3) \ 
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Survey of Perceptions 
Item 


Responses from preliminary 
field test-Consumer Group 


Responses from 
Distance Education 
Practitioner Grouo (N = 8) 


Is the content of the 
videotape too easy, too 
difficult, or just right? 


About right (1) 
Too easy (3) 
No response (4) 


About right (3) 
Too easy (2) 
No response (3) 


How useful would you have 
found this information prior to 
your first distance education 
class? 


Very necessary (7) 
Not applicable (1) 


Useful (5) 

Useful but need more 
examples (2) 
No response (1) 


List the topics which you 
judge to be the most helpful? 


All topics (1) 

Classroom management (2) 
Technology (2) 
Techniques/methods (2) 
Regular/distance ed contrast 
(1) 


Regular/distance ed contrast 
(2) 

Classroom management (3) 
Technology (1) 
No response (2) 


List the topics which you 
judge to be the least helpful? 


None (7) 

Lack of showing student 
interaction (1) 


Everything OK (2) 
Not enough pedagogy (2) 
Too much with same 
instructor (2) 
No responses (2) 


What topics might have been 
included in the videotape but 
were not? 


Techniques, content delivery, 
& technology (3) 
Uve interview footage (1) 
No response (4) 


More examples with different 
instructors/techniques (3) 
Specific good vs. specific bad 
examples (1) 

Operation of Technology (2) 
No response (2) 


Comments: 


Student remarks might be 
included (1) 

Transition between Parts (1) 
Freeze frame monologue 
needs to be improved (1 ) 
No responses (5) 


Nice quality video (1) 
Advance organizer needed (2) 
Vignettes needed for both 
men and women teaching (1) 
Regular ed/distance ed 
contrast not strong enough 
(1) 

Example does not generalize 
(1) 

No response (2) 
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When both groups were asked, "What skills that are illustrated in the videotape do you already 
possess?", all of the Consumer group responded that they possessed the regular classroom skills but 
none of the corresponding distance education skills. In contrast, seven of the Distance Education 
Practitioner group stated that they possessed both the regular classroom skills as well as the distance 
education skills. This information, which was a preliminary field test for the Consumer group and a 
content validation procedure for the Practitioner group, supported the purposes which had been defined 
for the two groups. 

Characteristics of the prototype version judged positively by a majority of respondents 

• A majority of the respondents in both groups stated that they found the initial 
motivation material interesting. 

• A majority of the respondents in both groups found the objectives clear and interesting 
as well as relevant. 

• A majority of the respondents in both groups judged that they would have found the 
information useful prior to their first distance education class. 

• A majority of the Consumer group judged the clarity of the examples adequate. 

• A majority of the Consumer group judged the vocabulary as adequate. 

Characteristics of the prototype version which need to be changed 

• An advance organizer is needed to introduce the tape The context at the beginning 
of the videotape needs to be stated and the purpose of the videotape needs to be 
explicitly stated. This change was evident from members of the Practitioner group who 
judged that motivational and interest levels needed enhancement. Two members of 
the Practitioner group failed to identify the purpose of the videotape and were unsure 
about the initial motivational material. 
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Similarities and differences between ragular classroom taaching techniques and 
distance education teaching techniques need to be explicitly stated for each segment 
of tha videotape. This change was identified from members of the Practitioner group 
who stated that the differences between regular classroom techniques and distance 
education techniques were unclear. 

More examples of different instructors and different content need to be added. This 
change comes from general comments from members of the Practitioner group and 
from comments addressing suggestions for improvement. 

Instructor comments to be expressed more enthusiastically. General comments from 
the Practitioner group included the recommendation that the voice-overs needed to be 
re-taped. 

Improve the legibility of the graphics. Two members of the Practitioner group stated 
that they had trouble reading the labels. 

Identify jargon and find substitute language. Two member of the Practitioner group 
stated that too much jargon was used when they were asked to comment about 
vocabulary usage. 

Make corrections to instances of inaccuracies. 

1 . discrepancy between elementary and middle school lessons 

2. instructor walked up to the monitor instead of camera 

The need for these corrections was pointed out by some of the reviewers. 
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Next steps 

In the developmental cycle, the next two sequential steps planned are (1) to revise the 
prototype version of the videotape following conclusions reached from this survey of perceptions and 
(2) to conduct a field test of the revised version of the videotape to members of the target population 
with emphasis given to the adaptation of instructional techniques to a distance education setting. 
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Appendix 



Perceptions from Viewing Adapting Instructional Techniques 



Directions: This survey follows a formative evaluation model similar to that described by Dick and 
Carey (1990). Please write your responses in the space provided following each question after you 
have reviewed the videotape. 

1. Is the initial motivation material interesting? 

2. Are the objectives clear and interesting as well as relevant? 

3. What skills that are illustrated in the videotape do you already possess? 

4. Does the videotape provide adequate: 



vocabulary? 



pacing? 



chunk size? 



clarity of descriptions? 



clarity of examples? 



sequence of content and activities? 



interest level? 



motivation? 



whether content was too easy, too difficult or just right? 



5. 



How useful would you have found this information prior to your first distance education 
class? 



6. 



List the topics which you judge to be the most helpful? 



7. 



List the topics which you judge to be the least helpful? 



8. 



What topics might have been included in the videotape but were not? 



Comments: 



